Tiered storage pool management and control for loosely coupled multiple storage environment

ABSTRACT

A system comprises a first storage system including a first storage controller, which receives input/output commands from host computers and provides first storage volumes to the host computers; and a second storage system including a second storage controller which receives input/output commands from host computers and provides second storage volumes to the host computers. A first data storing region of one of the first storage volumes is allocated from a first pool by the first storage controller. A second data storing region of another one of the first storage volumes is allocated from a second pool by the first storage controller. A third data storing region of one of the second storage volumes is allocated from the first pool by the second storage controller. A fourth data storing region of another one of the second storage volumes is allocated from the second pool by the second storage controller.

This application is a divisional of U.S. application Ser. No.12/860,086, filed Aug. 16, 2010, now U.S. Pat. No. 8,356,147, the entirecontent of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

The present invention relates generally to storage systems and, moreparticularly, to tiered storage pool management and control for loosecoupled multiple heterogeneous storage environment. The loosely coupledconnection uses standardized network and not-wide (narrower thaninternal bus) network, and control of each device connected to thenetwork is independent from that of other devices. By using the looselycoupled connection, one can consolidate heterogeneous (multi-vender,multi-type of product, and multi-generation of product) devices to onesystem.

Storage virtualization is a technology that is widely used. Examplesinclude thin provisioning and page-based tiered storage. The use ofstorage virtualization technology can improve storage usability andefficiency. The improvement of usability is by concealment of theinternal configuration (e.g., disk configuration, tier configuration,data location and so on). The improvement of efficiency is by sharing ofthe storage resources (e.g., disk workload, processor workload, internalbus workload, capacity and so on). Generally speaking, these storagevirtualization techniques can be applied only in one storage controller.One exception is the case involving the “Mount of external volume”function, wherein the external volume mount function is suitable for thetiered management situation between a high performance storage subsystemand a low cost storage subsystem. However, it will cause a waste ofresource and unnecessary overhead for performance by connecting a highperformance storage subsystem to a high performance storage subsystem.Many users have a plurality of storage subsystems in their IT(Information Technology) system. Each storage subsystem is independentfrom the other storage subsystems. This causes difficult and troublesomestorage system management for the system design and system operation.The result is a waste of resources in the overall storage system due tothe difficulty of resource sharing between two high performancestorages. It is difficult to manage the total resource amount andimprove the efficiency in the environment involving multiple storagesubsystems.

BRIEF SUMMARY OF THE INVENTION

Exemplary embodiments of the invention provide methods and apparatusesto improve the manageability of the overall storage system, in which theintegrated storage pool management shows the resource amount of eachtier. The integrated/system storage pool includes a plurality of storagesubsystems. The amount of each tier is the sum of the tier capacity ofthe pool in each storage subsystem. To improve the control of thestorage system, each storage subsystem manages its own resources. If astorage subsystem needs to use the resource of another storagesubsystem, that storage subsystem accesses the resource with standardI/O interface. The storage subsystem can perform tier management controlwith the external resource, and the priority of the tier of the externalresource depends on the network and external storage performance. Toimprove the efficiency of the storage resources, the integrated storagemanagement can select a way to rebalance among a plurality of storagesubsystems and the storage subsystem changes the assignment of resourcesand/or data according to the selection. Examples include lendingcapacity between storage subsystems, online volume migration betweenstorage subsystems, and changing path to external volumes betweenstorage subsystems. To avoid the circulation of resource allocation, thestorage subsystem forbids allocating a page on the other storagesubsystem resource for another storage subsystem, and the storagesubsystem does not show a user the resource amount lending form theother storage subsystem.

In accordance with an aspect of the present invention, a systemcomprises a first storage system including a first storage controller,which receives input/output (I/O) commands from host computers andprovides first storage volumes to the host computers; and a secondstorage system including a second storage controller which receives I/Ocommands from host computers and provides second storage volumes to thehost computers. A first data storing region of one of the first storagevolumes is allocated from a first pool by the first storage controller.A second data storing region of another one of the first storage volumesis allocated from a second pool by the first storage controller. A thirddata storing region of one of the second storage volumes is allocatedfrom the first pool by the second storage controller. A fourth datastoring region of another one of the second storage volumes is allocatedfrom the second pool by the second storage controller.

In some embodiments, allocation of the second data storing region ofsaid another one of the first storage volumes from the first pool by thesecond storage controller is prohibited. In response to allocation ofthe second data storing region of said another one of the first storagevolumes from the second pool by the first storage controller, the secondstorage controller subtracts an amount allocated to the first volumefrom a capacity of the second pool. In response to allocation of thesecond data storing region of said another one of the first storagevolumes from the second pool by the first storage controller, the secondstorage controller subtracts an amount allocated to the first volumefrom a capacity of the second pool. The first pool is managed using aplurality of first tiers; the second pool is managed using a pluralityof second tiers; and a storage media allocated to the second pool, whichwas allocated to second data storing region, is managed as anindependent tier from the plurality of first tiers. The independent tieris determined by network characteristics and access speed of the storagemedia used. If the storage media is same as a storage media used for thefirst pool, the storage media used for the first pool has a higherpriority as compared to the independent tier.

In one embodiment, a management server selects to lend capacity from thesecond pool to the first storage system or change control from thesecond storage controller to the first storage controller against astorage media which is coupled to both the first and second storagecontrollers, based on load requirement of the system. In anotherembodiment, a management server selects to lend capacity from the secondpool to the first storage system, migrate data from the first storagevolume to the second storage volume, or change control from the secondstorage controller to the first storage controller against a storagemedia which is coupled to both the first and second storage controllers,based on load requirement of the system.

In accordance with another aspect of the invention, a system comprises afirst storage system including a first storage controller, whichreceives input/output (I/O) commands from host computers and providesfirst storage volumes to the host computers; a second storage systemincluding a second storage controller, which receives I/O commands fromhost computers and provisions second storage volumes to the hostcomputers; and a storage media which is allocated to a first pool. Afirst data storing region of one of the first storage volumes isallocated from the first pool by the first storage controller. A seconddata storing region of one of the second storage volumes is allocatedfrom a second pool by the second storage controller. In response to acommand from a management server, the second storage controllerallocates the storage media to the second pool and the first storagecontroller releases the storage media from the first pool.

In specific embodiments, a management server selects to migrate a firststorage volume to a second storage volume, or change control from thesecond storage controller to the first storage controller against astorage media which is coupled to both the first and second storagecontrollers, based on load requirement of the system.

In accordance with another aspect of this invention, a system comprisesa plurality of storage systems, each including a storage controllerwhich receives input/output (I/O) commands from a host computer andprovisions storage volumes from an individual pool; and a managementserver coupled to the plurality of storage systems. The individual poolof each of the plurality of storage systems is allocated to a pluralityof storage media and comprises a plurality of individual tiers. Themanagement server gathers information of storage media allocated to theplurality of storage systems, and dynamically determines a plurality ofsystem tiers for a system pool, which includes storage media allocatedto the plurality of storage systems, based on characteristics of thestorage media in the gathered information.

In some embodiments, the management server informs determination resultsregarding the plurality of system tiers for the system pool to theplurality of storage systems. After a storage media is added to thesystem pool, the management server adjusts the plurality of individualtiers for a relevant individual pool of one of the plurality of storagesystems which is affected by addition of the added storage media. Inresponse to a user request, the management server changes the pluralityof system tiers for the system pool. In response to a change of theplurality of system tiers for the system pool, the plurality ofindividual tiers are automatically adjusted. The management serverdetermines at least one of configuration or location of the storagevolumes based on conditions of the storage controllers for the pluralityof storage systems.

In specific embodiments, in response to a request to release theindividual pool of one of the storage systems, at least one of thefollowing occurs in the system: internal storage volumes of theindividual pool to be released are migrated to the individual pool ofanother storage system of the plurality of storage systems; storagemedia allocated and directly controlled by the storage controller of thefirst storage system are allocated to the storage controller of anotherstorage system of the plurality of storage systems; and data of externalvolumes, which are controlled by the storage controller of anotherstorage system of the plurality of storage systems, is migrated to astorage volume controlled by the storage controller of said anotherstorage system controlling the data of the external volumes. The data ofexternal volumes is migrated to an internal storage volume controlled bythe storage controller of said another storage system controlling thedata of the external volumes or is migrated to a storage volume withinthe individual pool of said another storage system controlling the dataof the external volumes.

In another aspect of the present invention, a system comprises: a firststorage system including a first storage controller, which receivesinput/output (I/O) commands from host computers and provides firststorage volumes to the host computers; a second storage system includinga second storage controller, which receives I/O commands from hostcomputers and provisions second storage volumes to the host computers;and a storage media which is allocated to a first pool. A first datastoring region of one of the first storage volumes is allocated from thefirst pool by the first storage controller and a second data storingregion of one of the second storage volumes is allocated from a secondpool by the second storage controller. In response to a command from amanagement server, the second storage controller allocates the storagemedia to the second pool and the first storage controller releases thestorage media from the first pool. The system further comprises amanagement server which selects to migrate a first storage volume to asecond storage volume, or change control from the second storagecontroller to the first storage controller against a storage media whichis coupled to both the first and second storage controllers, based onload requirement of the system.

These and other features and advantages of the present invention willbecome apparent to those of ordinary skill in the art in view of thefollowing detailed description of the specific embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a management view of an informationsystem in which the method and apparatus of the invention may beapplied.

FIG. 2 illustrates an example of the logical configuration of theinformation system of FIG. 1.

FIG. 3 illustrates the configuration of the tiered thin provisioningpool of each frontend storage subsystem and the relation between thefrontend storage subsystems.

FIG. 4 illustrates an exemplary configuration of the frontend storagesubsystem.

FIG. 5 illustrates an example of the Disk Management Table.

FIG. 6 illustrates an example of the Tier Management Table.

FIG. 7 illustrates an example of the Volume Management Table.

FIG. 8 illustrates an example of the relation of the tiered thinprovisioning pool, RAID groups, and tiers.

FIG. 9 illustrates an example of the relation of the host volumes, crossmount volumes, tiered thin provisioning, and pages.

FIG. 10 illustrates an example of the configuration of the integratedstorage pool management server.

FIG. 11 illustrates an example of the table structure of IntegratedStorage Pool Tier Management Table.

FIG. 12 illustrates an example of the table structure of the StorageSubsystem Management Table.

FIG. 13 illustrates an example of the table structure of the SystemInformation Table.

FIG. 14 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Expansion Control Program.

FIG. 15 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Reduction Control Program.

FIG. 16 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Tier Configuration Control Program.

FIG. 17 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Volume Control Program.

FIG. 18 is an example of a flow diagram illustrating the process ofIntegrated Storage Pool Rebalancing Control Program.

FIG. 19 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Monitor Program.

FIG. 20 shows an example of the Integrated Storage Pool Element Manageras an input/output (I/O) interface for a user.

FIG. 21 shows an example of the Integrated Storage Pool Tier Manager asan I/O interface for a user.

FIG. 22 shows an example of the Integrated Storage Pool Volume Manageras an I/O interface for a user.

FIG. 23 illustrates the relation between tiers in the thin provisioningpools and a tier in the integrated storage pool.

FIG. 24 shows an example of the sequence of installation of storagesubsystems and expansion of the integrated storage pool.

FIG. 25 shows the state of the system after procedure S1-5 in FIG. 24.

FIG. 26 illustrates the flow of data, information, and order to achievethe status in FIG. 25.

FIG. 27 shows the state of the system after procedure S1-9 in FIG. 24.

FIG. 28 shows the flow of data, information, and order to achieve thestatus in FIG. 27.

FIG. 29 shows an example of the sequence of volume provisioning from theintegrated storage pool.

FIG. 30 shows the state of the system after the sequence in FIG. 29.

FIG. 31 shows the flow of data, information, and order to achieve thestatus in FIG. 30.

FIG. 32 shows an example of the sequence of I/Os from the host to thehost volume.

FIG. 33 shows the state of the system after the write I/O in FIG. 32.

FIG. 34 shows the flow of data, information, and order to achieve thestatus in FIG. 33.

FIG. 35 shows an example of the sequence of tiered storage management.

FIG. 36 shows the state of the system during page migration in FIG. 35.

FIG. 37 shows the flow of data, information, and order to find necessityof tier alternation.

FIG. 38 show the flow of data, information, and order to migratecapacity pages to change the tier.

FIG. 39 shows an example of the sequence of resource informationgathering from each frontend storage subsystem.

FIG. 40 shows the flow of data, information, and order to migratecapacity pages to change the tier.

FIG. 41 shows an example of the sequence of changing the mount ofexternal volume between frontend storage subsystems.

FIG. 42 shows the state of the system during the remount process in FIG.41.

FIG. 43 shows the flow of data, information, and order to remount theexternal volume.

FIG. 44 shows an example of the sequence of increment for mountingcapacity from another frontend storage subsystem

FIG. 45 shows the state of the system during the remount process in FIG.44.

FIG. 46 shows the flow of data, information, and order to remount theexternal volume.

FIG. 47 shows an example of the sequence of volume migration between twostorage subsystems.

FIG. 48 shows the state of the system during the migration process inFIG. 47.

FIG. 49 shows the flow of data, information, and order to remount anexternal volume.

FIG. 50 shows an example of the sequence of reduction of the integratedstorage pool by removal of a storage subsystem.

FIG. 51 shows the state of the system during the reduction process ofthe integrated storage pool in FIG. 50.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the invention, reference ismade to the accompanying drawings which form a part of the disclosure,and in which are shown by way of illustration, and not of limitation,exemplary embodiments by which the invention may be practiced. In thedrawings, like numerals describe substantially similar componentsthroughout the several views. Further, it should be noted that while thedetailed description provides various exemplary embodiments, asdescribed below and as illustrated in the drawings, the presentinvention is not limited to the embodiments described and illustratedherein, but can extend to other embodiments, as would be known or aswould become known to those skilled in the art. Reference in thespecification to “one embodiment,” “this embodiment,” or “theseembodiments” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the invention, and the appearances ofthese phrases in various places in the specification are not necessarilyall referring to the same embodiment. Additionally, in the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the present invention. However,it will be apparent to one of ordinary skill in the art that thesespecific details may not all be needed to practice the presentinvention. In other circumstances, well-known structures, materials,circuits, processes and interfaces have not been described in detail,and/or may be illustrated in block diagram form, so as to notunnecessarily obscure the present invention.

Furthermore, some portions of the detailed description that follow arepresented in terms of algorithms and symbolic representations ofoperations within a computer. These algorithmic descriptions andsymbolic representations are the means used by those skilled in the dataprocessing arts to most effectively convey the essence of theirinnovations to others skilled in the art. An algorithm is a series ofdefined steps leading to a desired end state or result. In the presentinvention, the steps carried out require physical manipulations oftangible quantities for achieving a tangible result. Usually, though notnecessarily, these quantities take the form of electrical or magneticsignals or instructions capable of being stored, transferred, combined,compared, and otherwise manipulated. It has proven convenient at times,principally for reasons of common usage, to refer to these signals asbits, values, elements, symbols, characters, terms, numbers,instructions, or the like. It should be borne in mind, however, that allof these and similar terms are to be associated with the appropriatephysical quantities and are merely convenient labels applied to thesequantities. Unless specifically stated otherwise, as apparent from thefollowing discussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing,” “computing,”“calculating,” “determining,” “displaying,” or the like, can include theactions and processes of a computer system or other informationprocessing device that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system's memories or registers or otherinformation storage, transmission or display devices.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may include one or more general-purposecomputers selectively activated or reconfigured by one or more computerprograms. Such computer programs may be stored in a computer-readablestorage medium, such as, but not limited to optical disks, magneticdisks, read-only memories, random access memories, solid state devicesand drives, or any other types of media suitable for storing electronicinformation. The algorithms and displays presented herein are notinherently related to any particular computer or other apparatus.Various general-purpose systems may be used with programs and modules inaccordance with the teachings herein, or it may prove convenient toconstruct a more specialized apparatus to perform desired method steps.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the invention as described herein. The instructions of theprogramming language(s) may be executed by one or more processingdevices, e.g., central processing units (CPUs), processors, orcontrollers.

Exemplary embodiments of the invention, as will be described in greaterdetail below, provide apparatuses, methods and computer programs fortiered storage pool management and control for lose coupled multiplestorage environment.

Various aspects of this invention make use of the following storagesystem technologies: U.S. Pat. No. 6,823,442 to Cameron on thinprovisioning; U.S. Pat. No. 7,711,896 to Yamamoto et al. on mountingexternal volumes; U.S. Publication No. US2007/0055713 by Nagai et al. onpage-based tiered storage management; U.S. Patent Publication No.US2010/0070722 by Otani et al. on non-disruptive volume migration; andU.S. Patent Publication No. US2008/0184000 by Kawaguchi on mappinginformation migration between two storage modules. The entiredisclosures of these are incorporated herein by reference.

A. Management View of the System

FIG. 1 illustrates an example of a management view of an informationsystem in which the method and apparatus of the invention may beapplied. The figure shows a summary of the configuration andfunctionality of the system. The information system includes a pluralityof frontend storage subsystems 100 for storing data, a plurality ofbackend storage subsystems 200 for storage data, and a plurality ofhosts 500. Each host 500 connects one or more host volumes 131 and sendsI/Os to the host volumes 131. Some host volumes 131 may be shared by aplurality of hosts 500. Each host volume 131 provides a data storagearea, and is provisioned from an integrated or system storage pool 600.The data of the host volume 131 is stored in the frontend storagesubsystems 100 and/or backend storage subsystems 200.

The integrated storage pool 600 includes the frontend storage subsystems100 and backend storage subsystems 200. The integrated storage pool 600provides the host volumes 131 to the hosts 500. The relation between thehost volumes 131 and the storage in which the data is located isconcealed from the user by the integrated storage pool 600. Theintegrated storage pool 600 can add one or more frontend storagesubsystems 100 and/or one or more backend storage subsystems 200 ondemand. It can also remove one or more frontend storage subsystems 100and/or one or more backend storage subsystems 200 on demand unless theresource of the integrated storage pool 600 is lacking after theremoval. The integrated storage pool 600 can balance its own resourceamong the frontend storage subsystems 100 and backend storage subsystems200. The balancing process is hidden from the hosts 500. As discussed inmore detail below, the management server 300 gathers information ofstorage media allocated to the plurality of storage subsystems, anddynamically determines a plurality of system tiers for the system pool600, which includes storage media allocated to the plurality of storagesubsystems, based on characteristics of the storage media in thegathered information. As used herein, “dynamically” means that duringthe online operation by the management server 300 (e.g., in determiningthe system tiers), the plurality of storage subsystems are capable ofreceiving I/O requests from the host computer(s).

B. Configuration of the System

B1. Logical Configuration of the System

FIG. 2 illustrates an example of the logical configuration of theinformation system of FIG. 1. The information system includes thefrontend storage subsystems 100, the backend storage subsystems 200, theintegrated storage pool management server 300, the networks 400 and 401,and the hosts 500.

The frontend storage subsystem 100 stores data. The data is in a tieredthin provisioning pool 110 as an individual pool. The frontend storagesubsystem 100 provides host volumes 131 to the hosts 500 and providescross mount volume 132 to other frontend storage subsystems 100. Thefrontend storage subsystem 100 is connected to the hosts 500 and theintegrated storage pool management server 300. The frontend storagesubsystem 100 can connect to the backend storage subsystems 200 and cantransfer data to/from the backend storage subsystems 200 via the network401. The detail explanation is provided in FIG. 4 as described below.The backend storage subsystem 200 is connected to the frontend storagesubsystems 100, and provides backend volumes 231 to the frontend storagesubsystems 200.

The integrated storage pool management server 300 connects to thefrontend storage subsystems 100, the backend storage subsystems 200, andthe hosts 500. The integrated storage pool management server 300 managesthe storage resource and the configuration of the frontend storagesubsystems 100 and backend storage subsystems 200, as well as theconnection between the hosts 500 and the frontend storage subsystems100. The detail explanation is provided in FIG. 10 as described below.

The network 400 transfers data, information, and order among the hosts500, frontend storage subsystems 100, and integrated storage poolmanagement server 300. The network 401 transfers data, information, andorder among the frontend storage subsystems 100, backend storagesubsystems 200, and integrated storage pool management server 300. Thenetwork 400 and network 401 can be consolidated. The hosts 500 areconnected to the frontend storage subsystems 100. The path configurationfor each host 500 is defined in the Path Management Module 501 of thehost 500. The I/O path follows to the configuration.

FIG. 3 illustrates the configuration of the tiered thin provisioningpool 110 of each frontend storage subsystem and the relation between thefrontend storage subsystems 100. The tiered thin provisioning pool 110includes one or more RAID groups 112. The RAID groups 112 can becategorized into one or more types of tier, such as RAID groups 112-1,112-2, 112-3, 112-4, 112-6 and so on. Each type of RAID groups hascertain characteristics in reliability, availability, performance,capacity, cost, connection protocol, and physicality. Examples includeRAID Level, RPMs of HDD, type of media, location (internal/external) andso on. The frontend storage subsystem 100 can use volumes on an externalstorage subsystem as its own RAID group by an external volume mountfunction. As seen in FIG. 3, the backend volume 231 is mounted as RAIDgroup 112-5 by the external volume mount function, and the cross mountvolume 132 is mounted as RAID group 112-6 by the external volume mountfunction. The tiered thin provisioning pool 110 is divided to aplurality of tiers. The details of the configuration and control methodare described in FIG. 8 and FIG. 9 as described below.

The host volume 131 and cross mount volume 132 are provisioned from thetiered thin provisioning pool 110 with tier configuration. The hostvolume 131 is connected by the host 500, and the cross mount volume 132is connected by another frontend storage subsystem 100.

B2. Physical Configuration of Frontend Storage Subsystem

FIG. 4 illustrates an exemplary configuration of the frontend storagesubsystem 100. It includes a storage controller 101 and disks 106. Thestorage controller 101 has a CPU 102, a memory 103, host interfaces 104,disk interfaces 105, a management server interface 107, and disks 106.The CPU 102 controls the storage subsystem using the control programsand control tables in the memory 103. The memory 103 stores data,programs, and tables, and includes cache area 103-0, Disk ManagementTable 103-1, Tiered Pool Management Table 103-2, Volume Management Table103-3, Page Management Table 103-4, Performance Monitor Program 103-5,Volume I/O Control Program 103-6, Disk I/O Control Program 103-7, PageControl Program 103-8, and Volume Migration Control Program 103-9.

The memory 103 has a cache area 103-0 for storing read and write cachedata. The Disk Management Table 103-1 manages the configuration of theRAID groups (see FIG. 5). The Tiered Pool Management Table 103-2 managesthe configuration of the tiers in the tiered thin provisioning pool 110(see FIG. 6). The Volume Management Table 103-3 manages the setting ofthe host volume 131 and cross mount volume 132 (see FIG. 7). The PageManagement Table 103-4 manages the page allocation of the thinprovisioning function. The Performance Monitor Program 103-5 gathers thestorage configuration and workload information and sends them to theintegrated storage pool management server 300. The Volume I/O ControlProgram 103-6 controls read and write I/Os from the hosts 500 andfrontend storage subsystem 100. The Disk I/O Control Program 103-7controls read and write I/Os to the disks 106, frontend storagesubsystem 100, and backend storage subsystem 200. The Page ControlProgram 103-8 controls the allocation and reallocation of thinprovisioning pages. When write I/O occurs, the page allocation processruns. When resource unbalancing occurs, the page reallocation processwill run to rebalancing. When removal of RAID groups occurs, the pagereallocation process will also run to cause the page to leave theremoved RAID group. The Volume Migration Control Program 103-9 controlsthe volume data, volume configuration, and volume attribution migrationfrom/to the frontend storage subsystem 100 to migrate a volume.

The host interface 104 is an interface to connect to the hosts 500, andreceives host I/O from the host 500 and transfers it to the CPU 102. Thedisk interface 105 is an interface to connect to the disks 106, andsends disk I/O received from the CPU 102 to the disks 106. Themanagement server interface 107 is an interface to connect to theintegrated storage pool management server 300. It sends storageinformation received from the CPU 102 to the integrated storage poolmanagement server 300, and receives orders from the integrated storagepool management server 300 and transfers them to the CPU 102.

The disks store storage data. There are multiple types of disks 106,including disks 106-1, disks 106-2, disks 106-3, and disks 106-4 as seenin FIG. 4. A plurality of disks 106 make up a RAID group. The RAID groupis RAID group 112 in FIG. 3.

B3. Table Structure of Frontend Storage Subsystem

FIG. 5 illustrates an example of the Disk Management Table 103-1, whichmanages the configuration of the RAID group 112 in the frontend storagesubsystem 100. The table has columns of RAID group ID 103-1-1, capacityinformation 103-1-2, attribution information 103-1-3, internal/externallocation information 103-1-4, RAID level information 103-1-5, and diskID 103-1-6. The RAID group ID 103-1-1 is a unique number in the frontendstorage subsystem 100. The capacity information 103-1-2 contains theusable capacity of the RAID group. The attribution information 103-1-3provides the type of the RAID group and stores the media type andconnection type of the RAID group. It indicates whether the RAID groupis an external volume. The configuration, such as the type of thestorage subsystem and the tier configuration of the mounted volume, ispassed on from the integrated storage pool management server 300. Thelocation information 103-1-4 shows information about the connection type(internal or external). The RAID level information 103-1-5 providesinformation on the RAID level. The disk ID includes the ID of each diskin the RAID group. If the disk is an internal disk, it stores the diskID of the frontend storage subsystem 100. If the disk is an externalvolume, it stores the volume information which can be identified in thesystem including, for example, WWN (World Wide Name) or LUN (LogicalUnit number).

FIG. 6 illustrates an example of the Tier Management Table 103-2, whichmanages the configuration of the tiers in the thin provisioning pool110. The table has columns of tier ID 103-2-1, total capacityinformation 103-2-2, used capacity information 103-2-3, tierconfiguration setting 103-2-4, RAID group ID 103-2-5, and tier prioritysetting 103-2-6. The tier ID 103-2-1 is a unique number in the tieredthin provisioning pool 110. The total capacity information 103-2-2provides the total capacity of the RAID groups which belong to thistier. The used capacity information 103-2-3 provides the allocated pagecapacity from the tier to the host volume 131 and cross mount volume132. The tier configuration setting 103-2-4 is the definition of thetier and contains information on the types of RAID groups which belongto the tier. The RAID group ID 103-2-5 contains the IDs of the RAIDgroups which belong to the tier. The selection is based on theinformation in the tier configuration setting 103-2-4. The tier prioritysetting 103-2-6 contains the configuration of tier priority. The PageControl Program 103-8 allocates and reallocates a page based on thisinformation and the Volume Management Table 103-3.

FIG. 7 illustrates an example of the Volume Management Table 103-3,which manages the setting of the host volume 131 and cross mount volume132. The table has columns of volume ID 103-3-1, volume capacity103-3-2, allocated capacity 103-3-3, usable tier setting 103-3-4, andvolume type information 103-3-5. The volume ID 103-3-1 is the ID of thevolume. The volume capacity 103-3-2 provides the capacity of the volume.The allocated capacity 103-3-3 is the total capacity of allocated pagesto the volume. The tier setting 103-3-4 is the tier from which thevolume can allocate page(s). If the volume is the cross mount volume132, it can prevent circular page allocation among a plurality offrontend storage subsystems 100 by preventing the cross mount volume 132from using tiers which have external volume(s) that another frontendstorage subsystem provides. The volume type information 103-3-5 containsdevice information which accesses the volume.

B4. Logical Configuration of Tiered Thin Provisioning Function

FIG. 8 illustrates an example of the relation of the tiered thinprovisioning pool 110, RAID groups 112, and tiers 111. The tiered thinprovisioning pool 110 includes a plurality of tiers 111. Each tier 111is a group of RAID groups which are listed in the RAID group ID 103-2-5.The selection of RAID group is based on the information in the tierconfiguration setting 103-3-4. FIG. 8 shows several tiers with differenttypes of media: Tier 111-1, Tier 111-2, Tier 111-3, Tier 111-4, Tier111-5, Tier 111-6 and so on. Each tier has a priority setting. Theinformation is stored in the tier priority setting 103-2-5.

FIG. 9 illustrates an example of the relation of the host volumes 131,cross mount volumes 132, tiered thin provisioning 110, and pages. When avolume uses a thin provisioning function, the volume allocates acapacity page 131-2 from the tiered thin provisioning pool 110 to avolume page 131-1 which covers the address where write I/O occurred. Therelation between a volume page 131-1 and a capacity page 131-2 is storedin the Page Management Table 103-4. The allocation can be changed bydata copy between an allocated capacity page 131-2 to another capacitypage and changing the mapping information between the volume page 131-1and the capacity page 131-2. The frontend storage subsystem 100automatically changes the allocation that depends on the performanceinformation, workload information, storage configuration, remaining tiercapacity, tier priority setting in the tier priority setting 103-2-5,and volume QoS setting in the volume type information 103-3-5.

B5. Physical Configuration of Integrated Storage Pool Management Server

FIG. 10 illustrates an example of the configuration of the integratedstorage pool management server 300. It includes a CPU 301, a systemmanagement interface 302, a memory 303, input devices 304, and a display305. The CPU 310 controls the server 300 by using the control programsand control tables in the memory 303. The system management interface302 is an interface to connect the frontend storage subsystem 100,backend storage subsystem 200, and hosts 500. It sends commands to thefrontend storage subsystem 100, backend storage subsystem 200, and hosts500 to change that configuration. It receives storage performance andworkload and configuration information from the frontend storagesubsystem 100 and backend storage subsystem 200.

The memory 303 stores data, programs, and tables. An Integrated StoragePool Tier Management Table 303-1 stores the tier configuration andstatus information of the integrated storage pool 600. The details ofthe structure are shown in FIG. 11. A Storage Subsystem Management Table303-2 stores the configuration and status information of each frontendstorage subsystems 100 and backend storage subsystems 200. The detailsof the structure are shown in FIG. 12. A System Information Table 303-3stores the configuration, performance, and workload information of ITdevices in the system except storage devices. The details of thestructure are shown in FIG. 13. An Integrated Storage Pool ExpansionControl Program 303-4 installs new frontend storage subsystem 100 andbackend storage subsystem 200 into the integrated storage pool 600. Thisprogram sets up the tier configuration to the frontend storage subsystem100 based on the information in the Integrated Storage Pool TierManagement Table 303-1, and sets up external volume mount and crossmount configuration. The details of the process are shown in FIG. 14. AnIntegrated Storage Pool Reduction Control Program 303-5 removes oldfrontend storage subsystem 100 and backend storage subsystem 200 fromthe integrated storage pool 600. This program orders the removedfrontend storage subsystem 100 to release the storage resources, andorders the non-removed frontend storage subsystems 100 to take over thedata and resources of the removed frontend storage subsystem 100. Thedetails of the process are shown in FIG. 15. An Integrated Storage PoolTier Configuration Control Program 303-6 sets up and/or changes the tierconfiguration of each frontend storage subsystem 100 based on theinformation in the Integrated Storage Pool Tier Management Table 303-1.The details of the process are shown in FIG. 16. An Integrated StoragePool Volume Control Program 303-7 selects the frontend storage subsystem100 and orders the frontend storage subsystem 100 to provision a volumewith QoS setting and host path setting. The details of the process areshown in FIG. 17. An Integrated Storage Pool Resource Control Program303-8 checks the resource to find unacceptable resource unbalance. Ifthe program finds unbalance, it orders the frontend storage subsystems100 to rebalance the resource in the integrated storage pool 600. Thedetails of the process are shown in FIG. 18. An Integrated Storage PoolMonitor Program 303-9 gathers configuration, performance, and workloadinformation, and alerts each frontend storage subsystem 100 and backendstorage subsystem 200.

The input devices 304 allow the user to input storage configurationsetting. The display 305 is an output device for the user, and a pointer305-0 shows the current input area to the user. The display 305 shows anumber of managers. An Integrated Storage Pool Element Manager 305-1presents a window as a user interface to order the installation of newfrontend storage subsystem 100 and backend storage subsystem 200 intothe integrated storage pool 600. The detailed image of the window isshown in FIG. 20. An Integrated Storage Pool Tier Manager 305-2 presentsa window as a user interface to set the configuration of tier of theintegrated storage pool 600. The detailed image of the window is shownin FIG. 21. An Integrated Storage Pool Volume Manager 305-3 presents awindow as a user interface to provision new host volume 131 and changethe configuration of existing host volume 131 of the integrated storagepool 600.

B6. Table Structure of Integrated Storage Pool Management Server

FIG. 11 illustrates an example of the table structure of IntegratedStorage Pool Tier Management Table 303-1. This table manages theconfiguration of each tier in the integrated storage pool 600. TheIntegrated Storage Pool Expansion Control 303-4 generates theconfiguration of tiers in the tiered thin provisioning pool 110 in a newfrontend storage subsystem 100 from this table at registration. TheIntegrated Storage Pool Volume Control 303-7 generates the configurationand location of the volume at volume provisioning.

The Integrated Storage Pool Tier Management Table 303-1 has thefollowing columns. The tier ID 303-1-1 is an ID of the tier which is aunique number in the integrated storage pool 600. The tier configurationsetting 303-1-2 provides definition of the tier, including the types oftiers which belong to the thin provisioning pools 110 in the frontendstorage subsystems 100. The user inputs these settings from theIntegrated Storage Pool Tier Manager 305-2. The internal tier prioritysetting 303-1-3 provides priority information of tier use when thefrontend storage subsystem 100 uses internal RAID groups or externalvolumes of the backend storage subsystems 200. The user inputs thesesettings from the Integrated Storage Pool Tier Manager 305-2. When theintegrated storage pool management server 300 changes or applies tiersetting to internal RAID group tier of the tiered thin provisioning pool110, the priority setting of the tier is based on this information andthe external tier priority setting 303-1-4. The external tier prioritysetting 303-1-4 provides priority information of tier use when thefrontend storage subsystem 100 uses external volumes of other frontendstorage subsystems 100. The integrated storage pool management server300 calculates this value with priority information in the internal tierpriority setting 303-1-3 and latency and workload information in theSystem Information Table 303-3. When the integrated storage poolmanagement server 300 changes or applies tier setting to the tier of thetiered thin provisioning pool 110 which consists of external volume tierfrom the frontend storage subsystem 100, the priority setting of thetier is based on this information and the external tier priority setting303-1-4.

The storage subsystem ID 303-1-5 provides the ID of the frontend storagesubsystems 100 which belongs to the integrated storage pool 600. Thisinformation is registered by the Integrated Storage Pool ExpansionControl 303-4. The thin provisioning pool tier ID 303-1-6 provides theID of tier in the thin provisioning pools 110 which relates to the tierof the integrated storage pool 600. This information is registered bythe Integrated Storage Pool Expansion Control 303-4. The storagesubsystem tier total capacity 303-1-7 provides the total capacity of thetier of the tiered thin provisioning pool 110 except the external volumetier of the frontend storage subsystems 100. This information isgathered and stored by the Integrated Storage Pool Monitor Program303-9. The storage subsystem tier used capacity 303-1-8 provides theused capacity of the tier of the tiered thin provisioning pool 110except the external volume tier of the frontend storage subsystems 100.This information is gathered and stored by the Integrated Storage PoolMonitor Program 303-9. The Storage Subsystem Tier Limit Workload 303-1-9provides the performance information of the tier of the tiered thinprovisioning pool 110. This information is calculated from theconfiguration of the tier such as, for example, numbers of disk, type ofmedia, RAID level, and the like. The configuration information isgathered and stored by the Integrated Storage Pool Monitor Program303-9. The Storage Subsystem Tier Current Workload 303-1-10 providesworkload information of the tier of the tiered thin provisioning pool110. This information is gathered and stored by the Integrated StoragePool Monitor Program 303-9.

FIG. 12 illustrates an example of the table structure of the StorageSubsystem Management Table 303-2. This table manages the configurationand status of the frontend storage subsystems 100, volumes 133 in thefrontend storage subsystems 100, the hosts 500 which connect to thevolumes, and the network 400 which links the hosts and the volumes. TheVolume Capacity Information 303-2-6, Volume Used Capacity Information303-2-7, Volume Workload Information 303-2-8, and Cache Hit RateInformation 303-2-9 are important to select volumes for resourcerebalancing among the frontend storage subsystems 100.

The Storage Subsystem Management Table 303-2 has the following columns.The storage ID 303-2-1 provides the ID of the frontend storage subsystem100 as a unique number in the system. The volume ID 303-2-2 provides theID of the host volume 131 as a unique number in the frontend storagesubsystem 100. The host ID 303-2-3 provides the ID of the host 500 whichconnects to the volume as a unique number in the system. The networkzone ID 303-2-4 provides the ID of network zone that the host and thevolume uses as a unique number in the system. The tier ID 303-2-5provides the ID of tier in the tiered thin provisioning pool 110 fromwhich the volume can allocate/relocate pages as a unique number in thetiered thin provisioning pool 110. The Volume Capacity Information303-2-6 provides capacity information of the volume, and is gathered andstored by the Integrated Storage Pool Monitor Program 303-9. The VolumeUsed Capacity Information 303-2-7 provides used capacity information ofthe volume, and is gathered and stored by the Integrated Storage PoolMonitor Program 303-9. The Volume Workload Information 303-2-8 providesWorkload information of the volume, and is gathered and stored by theIntegrated Storage Pool Monitor Program 303-9. The Cache Hit RateInformation 303-2-9 provides cache hit/miss information of the volume,and is gathered and stored by the Integrated Storage Pool MonitorProgram 303-9.

FIG. 13 illustrates an example of the table structure of the SystemInformation Table 303-3. This table stores information about the actualperformance and current workload of the system except storage. Thisinformation is useful for selecting the method to rebalance the storageresource among the frontend storage subsystems 100 and backend storagesubsystems 200.

The System Information Table 303-3 has the following columns. The index303-3-1 provides the index of the performance of the system. ThePerformance Information 303-3-2 provides information on actualperformance. This value can be informed by the user. The WorkloadInformation 303-3-3 provides current workload of the component. Thisvalue is gathered by the Integrated Storage Pool Monitor Program 303-9,inputted by the host 500 and network 400, or calculated.

B7. Process of Programs of Integrated Storage Pool Management Server

FIG. 14 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Expansion Control Program 303-4. This programruns when a user adds a new storage subsystem to the integrated storagepool 600. The operation instruction comes from the Integrated StoragePool Element Manager 305-1.

The program start at step 303-4-1. In step 303-4-2, the user providespool addition input for new storage subsystem information to the StoragePool Element Manager 305-1. In step 303-4-3, the program gathers theconfiguration information of the new storage subsystem (e.g., RAID groupconfiguration). The program refers to the Integrated Storage TierManagement Table 303-1 and categorizes the RAID groups in the newstorage subsystem to tiers. In step 303-4-4, the program generates theconfiguration of the new storage subsystem from the gatheredinformation. If the storage subsystem is suitable to the backend storagesubsystem 200, the program selects the frontend storage subsystems 100to connect. In step 303-4-5, the program applies the external volumesetting to the new storage subsystem. If the new storage subsystem is abackend Storage Subsystem 200, the program orders the frontend storagesubsystems 100 to mount volumes as external volumes and install to thetiered thin provisioning pools 110. In step 303-4-6, the program appliesthe generated tier configuration to the new storage subsystem. If thenew storage subsystem is a frontend storage subsystem 100, the programorders the new storage subsystem to set up the generated tier for itsthin provisioning pool 110. In step 303-4-7, the program applies thecross mount volume setting to all storage subsystems in the integratedstorage pool. If the new storage subsystem is a frontend storagesubsystem 100, the program orders to establish cross mount volumes 132with QoS setting, and orders the other frontend storage subsystems 100to mount the cross mount volumes 132 of the new frontend storagesubsystem 100 as external volumes. The program ends at step 303-4-8.

FIG. 15 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Reduction Control Program 303-5. This programruns when a user remove an existing storage subsystem from theintegrated storage pool 600. The operation instruction comes from theIntegrated Storage Pool Element Manager 305-1.

The program starts at step 303-5-1. In step 303-5-2, the user provides apool reduction input of the information of the removed storage subsystemto the Storage Pool Element Manager 305-1. In step 303-5-3, the programgathers the configuration information of the new storage subsystem(e.g., RAID group configuration and volume configuration). The programrefers to the Integrated Storage Tier Management Table 303-1 to know thecurrent configuration of the integrated storage pool 600. In step303-5-4, the program generates the configuration of each remainingstorage subsystem from the gathered information. In step 303-5-5, theprogram orders the removed storage subsystems and remaining storagesubsystems to migrate data among them. More specifically, data ismigrated from the reduced storage subsystem to the other storagesubsystems. The Volume Migration Control Program 103-9 and/or PageControl Program 103-8 will be run in the frontend storage subsystems 100by this operation. In step 303-5-6, the program orders the removedstorage subsystem to unmount the external volume if the removed storagesubsystem is the frontend storage 100 and it mounts volumes of thebackend storage subsystem 200. In step 303-5-7, the program orders theremaining storage subsystem(s) to mount volumes of the backend storagesubsystems 200 that the removed storage subsystem had mounted if theremoved storage subsystem is a frontend storage subsystem. In step303-5-8, the program orders the remaining storage subsystem(s) tounmount cross mount volumes provided by the removed storage subsystem ifthe removed storage subsystem is a frontend storage subsystem. The PageControl Program 103-8 will be run in the remaining storage subsystems bythis operation. In step 303-5-9, the user disconnects the removedstorage subsystem from the network and removes it. The program ends atstep 303-5-10.

FIG. 16 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Tier Configuration Control Program 303-6. Thisprogram runs when a user changes the tier configuration of theintegrated storage pool 600. The operation instruction comes from theIntegrated Storage Pool Tier Manager 305-2.

The program starts at step 303-6-1. In step 303-6-2, the program gathersthe configuration information of each storage subsystem. The programrefers to the Integrated Storage Tier Management Table 303-1 to obtainthe current configuration of the integrated storage pool 600. Theprogram shows a user the total information of the integrated storagepool 600. The amount of tier resource (e.g., capacity) of the integratedstorage pool 600 is the sum of the amount of related tiers in the tieredthin provisioning pool 110 of each frontend storage subsystems 100. Inthis way, the user can know the amount of total resource and remainingresource for each tier in the system. In step 303-6-3, the user inputsthe new tier configuration of the integrated storage pool 600 from theStorage Pool Tier Manager 305-2. In step 303-6-4, the program generatesthe configuration for the new setting, such as tier priority andcategorization. In step 303-6-5, the program orders each frontendstorage subsystems 100 to change the tier configuration of the tieredthin provisioning pool 110. The Page Control Program 103-8 will be runin the frontend storage subsystems 100 by this operation. The programends at step 303-6-6.

FIG. 17 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Volume Control Program 303-7. This program runswhen a user provisions new volume of changes configuration of existingvolumes in the integrated storage pool 600. The operation instructioncomes from the Integrated Storage Pool Volume Manager 305-3.

The program starts at step 303-7-1. It refers to the Integrated StorageTier Management Table 303-1 to obtain the current configuration of theintegrated storage pool 600. In step 303-7-2, the program shows a userthe total information of the integrated storage pool 600. The amount oftier resource (e.g., capacity) of the integrated storage pool 600 is thesum of amount of related tiers in the tiered thin provisioning pool 110of each frontend storage subsystems 100. In this way, the user can knowthe amount of total resource and remaining resource for each tier in thesystem. In step 303-7-3, the user inputs the new volume configuration ofthe integrated storage pool 600 from the Storage Pool Volume Manager305-3, including, for example, capacity of the volume, host whichconnects to the volume, and QoS settings. In step 303-7-4, the programchecks which operation occurs. If it is volume provisioning, the programproceeds to step 303-7-5. If it is volume reconfiguration, the programproceeds to step 303-7-7. In step 303-7-5, the program selects asuitable frontend storage subsystem 100 to provision a host volume 131.The selection is generated from the remaining resource, convertibilityof network, and requirement for the volume. This program generates theQoS setting (using tier setting) based on the input from the StoragePool Volume Manager 305-3 and current configuration of the selectedfrontend storage subsystem 100. In step 303-7-6, the program orders thefrontend storage subsystem 100 to provision a host volume 131 withcapacity and QoS settings. The program orders the host 500 to connect tothe provisioned host volume 131. In step 303-7-7, the program generatesthe QoS setting (using tier setting) based on the input from the StoragePool Volume Manager 305-3 and current configuration of the selectedfrontend storage subsystem 100. In step 303-7-8, the program orders thefrontend storage subsystem 100 to change capacity and QoS settings ofthe host volume 131. If the connecting host 500 is changed, the programorders the current host 500 to disconnect from the volume and orders theselected host to connect to the volume. The program ends at step303-7-9.

FIG. 18 is an example of a flow diagram illustrating the process ofIntegrated Storage Pool Rebalancing Control Program 303-8. This programruns periodically. The program starts at step 303-8-1. In step 303-8-2,the program refers to the Integrated Storage Tier Management Table 303-1and Storage Subsystem Management Table 303-2 to obtain the currentconfiguration and usage of the integrated storage pool 600. In step303-8-3, the program checks for unacceptable-unbalanced usage of tier inthe integrated storage pool 600. If it is balanced, the program ends atstep 303-8-11. Otherwise, in step 303-8-4, the program generates thefollowing rebalancing ways to resolve the unbalance.

One approach is the assignation of a backend external volume between twofrontend storage subsystems (see FIG. 39). Another is volume migrationbetween two frontend storage subsystems (see FIG. 46). Yet anotherapproach involves capacity lending by cross mount volume between twofrontend storage subsystems. (see FIG. 43). In step 303-8-5, the programselects the best way out of those in Step 303-8-4. The selection isdecided based on effect, system workload, and changing time. Theparameters are generated from the System Information Table 303-3,Integrated Storage Tier Management Table 303-1, and Storage SubsystemManagement Table 303-2. In step 303-8-6 (first selection), the programorders a resource-sufficient frontend storage subsystem 100 to unmountan external volume of a backend storage subsystem 200. By this unmountoperation, the Page Control Program 103-8 will run in theresource-absent frontend storage subsystem 100 to evacuate allocatedcapacity pages 131-2 to other RAID groups. In step 303-8-7, the programorders to a resource-absent frontend storage subsystem 100 to mount theunmounted volume as an external volume and to install the mounted volumeto a suitable tier. In step 303-8-8 (second selection), the programorders a resource-absent frontend storage subsystem 100 to migrate ahost volume 131 to a resource-sufficient frontend storage subsystem 100.By this migration operation, the Volume Migration Control Program 103-9will run in the resource-absent frontend storage subsystem 100 andresource-sufficient frontend storage subsystem 100. In step 303-8-9(third selection), the program orders a resource-sufficient frontendstorage subsystem 100 to provision a cross mount volume 132 to provide acapacity to a resource-absent frontend storage subsystem 100. In step303-8-10, the program orders a resource-sufficient frontend storagesubsystem 100 to mount the cross mount volume 132 which theresource-sufficient frontend storage subsystem 100 has provisioned. Theprogram ends at step 303-8-11.

FIG. 19 is an example of a flow diagram illustrating the process of theIntegrated Storage Pool Monitor Program 303-8. This program runsperiodically. The program starts at step 303-9-1. In step 303-9-2, theprogram receives the configuration and workload information from thefrontend storage subsystems 100 and backend storage subsystems 200. Instep 303-9-3, the program stores the received information to the SystemInformation Table 303-3, Integrated Storage Tier Management Table 303-1,and Storage Subsystem Management Table 303-2. The program ends at step303-9-4.

B8. Display Image of Integrated Storage Pool Management Server

FIG. 20 shows an example of the Integrated Storage Pool Element Manager305-1 as an input/output (I/O) interface for a user. Using this I/Ointerface, the user can input a new storage subsystem to be added to theintegrated storage pool 600, can input an existing storage subsystem tobe removed from the integrated storage pool 600, and can check thecurrent resource information and resource information after modificationof the integrated storage pool 600. The I/O interface includes thefollowing features. A Pool Tier Resource Information Panel 305-1-1 showsa user the current resource amount of each tier. When the user setsaddition and/or removal of a storage subsystem to/from the integratedstorage pool 600, the panel shows the resource information after themodification. A Pool Status Information Panel 305-1-2 shows the currentstatus of the integrated storage pool 600. A Pool ConfigurationInformation Panel 305-1-3 shows a user a list of storage subsystems inthe integrated storage pool 600. A user can select storage subsystemsfrom this list to remove from the integrated storage pool 600. A StorageSubsystem Information Panel 305-1-4 shows a user a list of storagesubsystems which is not in the integrated storage pool 600 but can beadded to it. A user can select storage subsystems from this list to addto the integrated storage pool 600. Using an Add Button 305-1-4, a usercan set addition of storage subsystems to the integrated storage pool600 by pushing this button after the selection of storage subsystemsfrom the Storage Subsystem Information Panel 305-1-4. Using a RemoveButton 305-1-5, a user can set removal of storage subsystems from theintegrated storage pool 600 by pushing this button after the selectionof storage subsystems from Pool Configuration Information Panel 305-1-3.Using an Apply Button 305-1-7, a user can send orders of the IntegratedStorage Pool Expansion Control Program 303-4 to add storage subsystemsto the integrated storage pool 600 by pushing this button; additionallyor alternatively, a user can send orders of the Integrated Storage PoolReduction Control Program 303-5 to add storage subsystems to theintegrated storage pool 600 by pushing this button. A user gets back theconfiguration to current state by pushing a Cancel Button 305-1-8.

FIG. 21 shows an example of the Integrated Storage Pool Tier Manager305-2 as an I/O interface for a user. The user can input tierconfiguration of the integrated storage pool 600 and can check thecurrent resource information and resource information after modificationof the integrated storage pool 600 with this interface. The interfaceincludes the following features. A Pool Tier Resource Information Panel305-2-1 shows a user the current resource amount of each tier. When theuser sets tier configuration change to the integrated storage pool 600,this panel shows the resource information after the modification. A PoolStatus Information Panel 305-1-2 shows the current status of theintegrated storage pool 600. A Tier Configuration Information Panel305-2-3 shows the user a list of tiers in the integrated storage pool600. The user can change the tier configuration of the integratedstorage pool 600. The user can send orders of the Integrated StoragePool Tier Configuration Control Program 303-6 to change the tierconfiguration of the integrated storage pool 600 by pushing an ApplyButton 305-2-7. The user can get back the configuration to currentstatus by pushing a Cancel Button 305-2-8.

FIG. 22 shows an example of the Integrated Storage Pool Volume Manager305-3 as an I/O interface for a user. The user can input volumeconfiguration of the integrated storage pool 600 and can check thecurrent resource information of the integrated storage pool 600 withthis interface. The interface includes the following features. A PoolTier Resource Information Panel 305-3-1 shows the user the currentresource amount of each tier. A Pool Status Information Panel 305-3-2shows the current status of the integrated storage pool 600. A VolumeConfiguration Information Panel 305-3-3 shows the user a list of volumesin the integrated storage pool 600. the user can change the volumeconfiguration of the integrated storage pool 600. The user can sendorders of the Integrated Storage Pool Volume Control Program 303-6 tochange the volume configuration of the integrated storage pool 600 bypushing an Apply Button 305-3-7 n. The user can get back theconfiguration to current status by pushing a Cancel Button 305-3-8.

B9. Relation Between Thin Provisioning Pool Tier and Integrated StoragePool Tier

FIG. 23 illustrates the relation between tiers in the thin provisioningpools 110 and a tier in the integrated storage pool 600. The integratedstorage pool 600 has tiers 611, which are categorized by the media type,RAID levels, cost, and so on. They are categorized to Tier 611-1, Tier611-2, Tier 611-3, Tier 611-4, Tier 611-5, and so on. The categorizationis based on the setting in the Integrated Storage Pool Tier ManagementTable 303-1. Each tier 611 in the integrated storage pool 600 has thetiers 111 of each thin provisioning pool 110 of the frontend storagesubsystem 100. Each tier 611 has priority settings which are from theIntegrated Storage Pool Tier Management Table 303-1.

C. Operational Procedure of the System

C1. Integrated Storage Pool Expansion

FIG. 24 shows an example of the sequence of installation of storagesubsystems and expansion of the integrated storage pool 600. Thesequence has the following procedures. In procedure S1-1, the userintroduces new frontend storage subsystems 100 a and 100 b and sets upRAID group configuration and network configuration of the frontendstorage subsystems 100 a and 100 b. By this procedure, the frontendstorage subsystems 100 a and 100 b change the configuration in S1-2. Theuser orders the integrated storage pool 600 to expand with the frontendstorage subsystems 100 a and 100 b. By this procedure, the integratedstorage pool 600 changes the configuration in S1-3. In procedure S1-2,the frontend storage subsystem 100 establishes RAID groups 112 and getsnetwork information and login to the network. In procedure S1-3, first,the integrated storage pool management server 300 orders the frontendstorage subsystems 100 a and 100 b, proceeding to S1-4, to mountexternal volumes of the backend storage subsystems 100, to establish thetiered thin provisioning pool 110, and to establish tiers in the tieredthin provisioning pool 110 and proceed to S-14. Next, the integratedstorage pool management server 300 orders the frontend storagesubsystems (proceeding to S1-6) to provision cross mount volumes 132from the tiered thin provisioning pool 110 with QoS setting. Next, theintegrated storage pool management server 300 orders the frontendstorage subsystems (proceeding to S1-6) to mount the provisioned crossmount volume 132 of another frontend storage subsystem 100 and proceedto S1-8.

In procedure S1-4, the frontend storage subsystem 100 establishes atiered thin provisioning pool 110. The frontend storage subsystem 100mounts designated external volumes of the backend storage subsystems 200and installs the external volumes to the tiered thin provisioning pool110. The frontend storage subsystem 100 sets the designatedconfiguration of tiers 111. In procedure S1-5, the RAID groups 112 inthe tiered thin provisioning pool 110 are categorized to the tiers 111.In procedure S1-6, the frontend storage subsystem 100 makes cross mountvolumes 132 with the designated QoS setting. In procedure S1-7, thecross mount volumes 132 are provisioned. In procedure S1-8, the frontendstorage subsystem 100 mounts the designated cross mount volumes 132 ofsome other frontend storage subsystem 100. In procedure S1-9, externalvolumes of the cross mount volume 132 are mounted as RAID groups.

FIG. 25 shows the state of the system after procedure S1-5. The externalvolumes 231 of the backend storage subsystem 200 are mounted by thefrontend storage subsystems 100. Each frontend storage subsystem 100 hasa tiered thin provisioning pool 110 with internal RAID groups 112 andmounted external volumes 231.

FIG. 26 illustrates the flow of data, information, and order to achievethe status in FIG. 25. The Integrated Storage Pool Expansion ControlProgram 303-4 obtains the current configuration of the integratedstorage pool 600 from the Storage Subsystem Management Table 303-2 andIntegrated Storage Pool Tier Management table 303-1. The IntegratedStorage Pool Expansion Control Program 303-4 generates the newconfiguration of the integrated storage pool 600 and stores the newconfiguration to each table. The Integrated Storage Pool ExpansionControl Program 303-4 orders the Tiered Pool Management Tables 103-2 toset the information of the tiered thin provisioning pool 110 as thegenerated new setting. The Integrated Storage Pool Expansion ControlProgram 303-4 orders the Disk Management Table 103-1 to set the mountedexternal volume as the generated new setting. The Integrated StoragePool Expansion Control Program 303-4 orders the volume 231 of thebackend storage subsystem 200 to allow the I/O from the mountingfrontend storage subsystem 100.

FIG. 27 shows the state of the system after procedure S1-9. The crossmount volumes 132 of the frontend storage subsystem 100 are provisionedfrom the tiered thin provisioning pool 110. The cross mount volumes 132of the frontend storage subsystem 100 are mounted by other frontendstorage subsystems 100.

FIG. 28 shows the flow of data, information, and order to achieve thestatus in FIG. 27. The Integrated Storage Pool Expansion Control Program303-4 obtains the current configuration of Integrated Storage Pool 600from the Storage Subsystem Management Table 303-2 and Integrated StoragePool Tier Management Table 303-1. The Integrated Storage Pool ExpansionControl Program 303-4 generates the new configuration of the integratedstorage pool 600 and stores the new configuration to the IntegratedStorage Pool Tier Management Table 303-1. The Integrated Storage PoolExpansion Control Program 303-4 orders the Volume Management Tables103-3 to set the provisioned cross mount volumes 132. The IntegratedStorage Pool Expansion Control Program 303-4 orders the Disk ManagementTables 103-1 to set the mounted cross mount volumes 132 of the otherfrontend storage subsystem 100. The Integrated Storage Pool ExpansionControl Program 303-4 orders the Tiered Pool Management Table 103-2 toset the tier of the mounted cross mount volumes 132.

C2. Integrated Storage Pool Volume Provisioning

FIG. 29 shows an example of the sequence of volume provisioning from theintegrated storage pool 600. The sequence includes the followingprocedures. In procedure S2-1, the user inputs volume configuration tothe integrated storage pool management server 300, including, forexample, capacity, QoS (Tier 611 selection), and connected hostconfiguration. In procedure S2-2, the integrated storage pool managementserver 300 selects the setting of network zone, suitable storagesubsystem, and suitable tier (Tier 111) from the current resource statusof each frontend storage subsystem 100. The integrated storage poolmanagement server 300 orders the frontend storage subsystems 100 a and100 b to provision host volumes 131 with configuration of the aforesaidgenerated configuration. The integrated storage pool management server300 orders the host 500 to mount the provisioned host volumes 131. Inprocedure S2-3, the frontend storage subsystem 100 transfers theconfiguration information of the tiered thin provisioning pool 110. Inprocedure S2-4, the frontend storage subsystem 100 establishes the hostvolumes 131 with the designated configuration. In procedure S2-5, thehost volumes 131 are provisioned. In procedure S2-6, the host 500establishes a volume connection with the designated host volume 131.

FIG. 30 shows the state of the system after the sequence in FIG. 29. Thehost volumes 131 of the frontend storage subsystem 100 are provisionedfrom the tiered thin provisioning pool 110. The host volumes 131 of thefrontend storage subsystem 100 are connected from the host 500.

FIG. 31 shows the flow of data, information, and order to achieve thestatus in FIG. 30. The Integrated Storage Pool Volume Control Program303-7 obtains the resource configuration from the Storage SubsystemManagement Table 303-2 and Integrated Storage Pool Tier Management303-1. The Integrated Storage Pool Volume Control Program 303-7 storesthe information of the provisioned host volume 131 to the StorageSubsystem Management Table 303-2. The Integrated Storage Pool VolumeControl Program 303-7 orders the Volume Management Tables 103-3 to setthe provisioned host volume 131. The Integrated Storage Pool ExpansionControl Program 303-4 orders the Path Management Module 501 to set theconnection to the host volumes 131.

C3. Host I/O

FIG. 32 shows an example of the sequence of I/Os from the host 500 tothe host volume 131. The sequence includes the following procedures. Inprocedure S3-1, the host 500 sends write I/O to the host volume 131 ofthe frontend storage subsystem 100. In procedure S3-2, the frontendstorage subsystem 100 receives the data. In procedure S3-3, the hostvolume 131 stores the received data to the cache area. In procedureS3-4, periodically, the frontend storage subsystem 100 checks the cachestatus, and if it finds the un-destaged data, the frontend storagesubsystem 100 stores the data to disks. If the frontend storagesubsystem 100 has not allocated a capacity page for the data yet, itallocates a new capacity page. In procedure S3-5, the host volume 131destages the data with capacity page allocation, if it is needed. Inprocedure S3-6, the RAID group 112 stores the write data to disks. Inprocedure S3-7, the host 500 sends read I/O to the host volume 131 ofthe frontend storage subsystem 100. In procedure S3-8, the frontendstorage subsystem 100 sends the data. In procedure S3-9, the host volume131 stages the read data from the disks to the cache area. In procedureS3-10, the RAID group 112 transfers the data from the disks to the cachearea.

FIG. 33 shows the state of the system after the write I/O in FIG. 32.The volume page 131-1 on the host volume 131 is allocated a capacitypage 131-2. The entity of the capacity page 131-2 is in the internalRAID group 112-4, external volume 231, or internal RAID group 112-4 ofanother frontend storage subsystem 100.

FIG. 34 shows the flow of data, information, and order to achieve thestatus in FIG. 33. The host 500 sends I/O request and data to the VolumeI/O Control Program 103-6. The Volume I/O Control Program 103-6 refersto Volume Management Table 103-3 to obtain the volume configuration. TheVolume I/O Control Program 103-6 stores the data into the cache area103-0. The Disk Control Program 103-7 checks the status of the cachearea 103-0 to find dirty data. The Disk Control Program 103-7 refers tothe Page Management Table 103-4 to judge whether the area needs toallocate a capacity page. If it does, the Disk Control Program 103-7refers to Tiered Pool Management Table 103-2 to find the tier from whicha capacity page should be allocated. It further stores the relationinformation between a volume page and a capacity page into the PageManagement Table 103-4. The Disk Control Program 103-7 finds disks andaddress information of the dirty data from the Page Management Table103-4 and Disk Management Table 103-1, and transfers the data to thedisks 106 or an external volume 231.

C4. Automated Tiered Storage Management in Frontend Storage Subsystem

FIG. 35 shows an example of the sequence of tiered storage management.In procedure S4-1, the frontend storage subsystem 100 checksperiodically the tier configuration of each allocated volume page. Itchanges the allocated tier subsequent to I/O workload alternation and/oruser requirement (i.e., changing of tier configuration). The method ofreallocation involves selecting a new capacity page, copying data oncapacity page from an existing capacity page to the newly allocatedcapacity page, and changing the mapping information on the PageManagement Table 103-4 to the newly allocated capacity page. Inprocedure S4-2, the RAID group 112 copies the data. In procedure S4-3,the host Volume 132 changes the mapping information of the volume page.

FIG. 36 shows the state of the system during page migration. The data onthe capacity page 131-2 a is copied to another capacity page 131-2 b.The volume page 131-1 changes the mapping from data on the capacity page131-2 a to data on the capacity page 131-2 b.

FIG. 37 shows the flow of data, information, and order to find necessityof tier alternation. The Performance Monitor Program 103-5 checksworkload and performance information from the Volume I/O Control Program103-6, cache area 103-0, Disk Control Program 103-7, and Tiered PoolManagement Table 103-2. The Volume I/O Control Program 103-6 providesthe I/O workload and I/O latency information and Read/Write rate. Thecache area 103-0 provides cache hit rate information. The Disk ControlProgram 103-7 provides disk workload and disk latency information. TheTiered Pool Management Table 103-2 provides tier status and usageinformation.

FIG. 38 show the flow of data, information, and order to migratecapacity pages to change the tier. The Performance Monitor Program 103-5notifies or instructs the Page Control Program 103-8 to start migration.The Page Control Program 103-8 allocates a new capacity page from asuitable tier by checking the Tiered Pool Management Table 103-2. Itorders the Disk Control Program 103-7 to copy the data between twocapacity pages and changes the mapping information on the PageManagement Table 103-4. The Disk Control Program 103-7 copies the data.It refers to the Disk Management Table 103-1 to obtain the configurationof the RAID groups. The copy process uses the cache area 103-0 totransfer the data. The Disk Control Program 103-7 stores data to thedisk 106 or external volume 231.

C5. Resource Information Gathering

FIG. 39 shows an example of the sequence of resource informationgathering from each frontend storage subsystem 100. In procedure S5-1,the integrated storage pool management server 300 periodically requestsresource information from each frontend storage subsystem 100. Inprocedure S5-2, the frontend storage subsystem 100 reports the resourceinformation to the integrated storage pool management server 300. Theinformation includes I/O workload (transaction and so on), tier usage(capacity, I/O locality and so on), and I/O performance (I/O rate, cachehit rate and so on).

FIG. 40 shows the flow of data, information, and order to migratecapacity pages to change the tier. The Performance Monitor Program 103-5reports workload and actual performance information to the IntegratedStorage Pool Monitor Program 303-1. The information includes I/Oworkload, I/O latency, read/write rate, hit rate, and disk latency. TheTiered Pool Management Table 103-2 reports resource usage information ofeach tier to Integrated Storage Pool Monitor Program 303-1. Theinformation includes tier status and tier usage. The Volume ManagementTable 103-3 reports resource usage information of each volume to theIntegrated Storage Pool Monitor Program 303-1. The information includesvolume status, volume usage, and volume workload.

C6. Remount an External Volume by Other Storage subsystem

FIG. 41 shows an example of the sequence of changing the mount ofexternal volume between frontend storage subsystems 100. This sequenceruns at step 303-8-6 and step 303-8-7. In procedure S6-1, the integratedstorage pool management server 300 selects external volumes 231 tochange the mounting frontend storage subsystem 100 b and selects theremounting frontend storage subsystem 100 a. The integrated storage poolmanagement server 300 orders the frontend storage subsystem 100 b toremove the external volumes 231. After the remove operation, theintegrated storage pool management server 300 orders the frontendstorage subsystem 100 a to mount the unmounted external volume 231 andto install its tiered thin provisioning pool 110. In procedure S6-2, thefrontend storage subsystem 100 b migrates the already-allocated capacitypages on the designated external volume 231 to other RAID groups. Inprocedure S6-3, the RAID group 132 which includes the external volumecopies data from the designated external volume 231 to other RAIDgroups. In procedure S6-4, the host volume 131 and cross mount volume132 change the mapping information of the copied capacity pages. Inprocedure S6-5, the frontend storage subsystem 100 b unmounts thedesignated external volume 231. In procedure S6-6, the unmountedexternal volume 231 leaves from the tiered thin provisioning pool 110.In procedure S6-7, the frontend storage subsystem 100 a mounts theunmounted external volume 231. In procedure S6-8, the mounted externalvolume 231 joins to the tiered thin provisioning pool 110.

FIG. 42 shows the state of the system during the remount process. Thefrontend storage subsystem 100 b migrates the capacity page 131-1 whichis on the RAID group 112-5 mounting the external volume 231 to anotherRAID group 112 and unmounts the external volume 231. The frontendstorage subsystem 100 a mounts the external volume 231 and sets it tothe tiered thin provisioning pool 110.

FIG. 43 shows the flow of data, information, and order to remount theexternal volume 231. The Integrated Storage Resource Control Program303-8 finds the unbalance of resource by checking the Storage SubsystemManagement Table 303-2. It considers the best way to rebalance byreferring to the Storage Subsystem Management Table 303-2, IntegratedStorage Tier Management Table 303-1, and System Information table 303-3.As a result, it selects the way to change the mount of an externalvolume from the frontend storage subsystem 100 b to the frontend storagesubsystem 100 a. The Integrated Storage Resource Control Program 303-8orders the Tiered Pool Management Table 103-2 on the frontend storagesubsystem 100 b to remove the RAID group 112 mounting the externalvolume 231 from the tiered thin provisioning pool 110. The IntegratedStorage Resource Control Program 303-8 orders the Disk Management Table103-1 on the frontend storage subsystem 100 b to unmount the externalvolume 231. The Page Control Program 103-8 on the frontend storagesubsystem 100 b allocates a new capacity page from a suitable tier bychecking the Tiered Pool Management Table 103-2. It orders the DiskControl Program 103-7 to copy the data between the two capacity pagesand changes the mapping information on the Page Management Table 103-4.

The Disk Control Program 103-7 on the frontend storage subsystem 100 bcopies the data by referring to the Disk Management table 103-1 toobtain the configuration of the RAID groups, using the cache area 103-0to transfer the data, and storing the data to the disk 106 or externalvolume 231. The Integrated Storage Resource Control Program 303-8 ordersthe external volume 231 to change the path from the frontend storagesubsystem 100 b to the frontend storage subsystem 100 a. The IntegratedStorage Resource Control Program 303-8 orders the frontend storagesubsystem 100 a and the Disk Management Table 103-1 to mount theexternal volume 231 as a RAID group 112. The Integrated Storage ResourceControl Program 303-8 orders the frontend storage subsystem 100 a andthe Tiered Pool Management Table 103-2 to make a RAID group 112 mountingexternal volume 231 join the tiered thin provisioning pool 110.

C7. Addition of Lending Capacity from Other Storage

FIG. 44 shows an example of the sequence of increment for mountingcapacity from another frontend storage subsystem 100. This sequence runsat step 303-8-9 and step 303-8-10. The sequence includes the followingprocedures. In procedure S7-1, the integrated storage pool managementserver 300 selects the frontend storage subsystem 100 b to lend thecapacity to the frontend storage subsystem 100 a. The integrated storagepool management server 300 orders the frontend storage subsystem 100 bto provision a cross mount volume 132. After the provisioning operation,the integrated storage pool management server 300 order the frontendstorage subsystem 100 a to mount the cross mount volume 132 and toinstall its tiered thin provisioning pool 110. In procedure S7-2, thefrontend storage subsystem 100 b provisions a cross mount volume 132from its tiered thin provisioning pool 110 with QoS setting for thecross mount volume 132. In procedure S7-3, the cross mount volume 132 isestablished. In procedure S7-4, the frontend storage subsystem 100 amounts the provisioned cross mount volume 132 of the frontend storagesubsystem 100 b as the RAID group 112-6, and installs the RAID group112-6 to the tiered thin provisioning pool 110. In procedure S7-5, theRAID group 112-6 mounting the cross mount volume 132 is joined to thetiered thin provisioning pool 110.

FIG. 45 shows the state of the system after the addition process. Thefrontend storage subsystem 100 a mounts a cross mount volume 132 of thefrontend storage subsystem 100 b. The frontend storage subsystem 100 auses the mounted cross mount volume 132 for resource of the tiered thinprovisioning pool 110.

FIG. 46 shows the flow of data, information, and order to remount theexternal volume 231. The Integrated Storage Resource Control Program303-8 finds the unbalance of resource by checking the Storage SubsystemManagement Table 303-2. It considers the best way to rebalance byreferring to the Storage Subsystem Management Table 303-2, IntegratedStorage Tier Management Table 303-1, and System Information table 303-3.As a result, it selects the way to lend the capacity from the frontendstorage subsystem 100 b to the frontend storage subsystem 100 a. TheIntegrated Storage Resource Control Program 303-8 orders the VolumeManagement Table 103-3 on the frontend storage subsystem 100 b toprovision a cross mount volume 231. The Integrated Storage ResourceControl Program 303-8 orders the Disk Management Table 103-1 on thefrontend storage subsystem 100 a to mount the cross mount volume on thefrontend storage subsystem 100 b and to join the mounted cross mountvolume 132 to the tiered thin provisioning pool 110.

C8. Volume Migration to Other Storage

FIG. 47 shows an example of the sequence of volume migration between twostorage subsystems. This sequence runs at step 303-8-8. In procedureS8-1, the integrated storage pool management server 300 selects hostvolumes 131 to migrate from the frontend storage subsystem 100 a to thefrontend storage subsystem 100 b. The integrated storage pool managementserver 300 orders to migrate from the frontend storage subsystem 100 ato the frontend storage subsystem 100 b. In procedure S8-2, the frontendstorage subsystem 100 b copies the configurations and data of thedesignated host volumes 131 on the frontend storage subsystem 100 a. Inprocedure S8-3, the frontend storage subsystem 100 a transfers the dataand configuration of the host volume 131. In procedure S8-4, the hostvolume 131 on the frontend storage subsystem 100 a transfers the data.In procedure S8-5, the host volume 131 on the frontend storage subsystem100 b is provisioned and stores the data. In procedure S8-6, the host500 changes the path from the host volume 131 on the frontend storagesubsystem 100 a to the host volume 131 on the frontend storage subsystem100 b. In procedure S8-7, the frontend storage subsystem 100 adisconnects the host path. In procedure S8-8, the frontend storagesubsystem 100 b connects the host path. In procedure S8-9, the frontendstorage subsystem 100 a deletes the migrated host volume 131. Inprocedure S8-10, the migrated host volume 131 releases the allocatedcapacity pages.

FIG. 48 shows the state of the system during the migration process. Thefrontend storage subsystem 100 a migrates a host volume 131 to thefrontend storage subsystem 100 b. The host 500 changes the path from thefrontend storage subsystem 100 a to the frontend storage subsystem 100b.

FIG. 49 shows the flow of data, information, and order to remount anexternal volume 231. The Integrated Storage Resource Control Program303-8 finds the unbalance of resource by checking the Storage SubsystemManagement Table 303-2. It considers the best way to rebalance byreferring to the Storage Subsystem Management Table 303-2, IntegratedStorage Tier Management Table 303-1, and System Information table 303-3.As a result, it selects the way to migrate an online volume from thefrontend storage subsystem 100 a to the frontend storage subsystem 100b.

The Integrated Storage Resource Control Program 303-8 orders the VolumeMigration Control Program 103-9 to migrate a host volume 131 to thefrontend storage subsystem 100 b. The Volume Migration Control Program103-9 on the frontend storage subsystem 100 a gathers the configurationof the host volume 131 from the Volume Management Table 103-3, TieredPool Management Table 103-2, and Page Management Table 103-4. It isparticularly important to know the allocation information from the PageManagement Table 103-4. Because this information is useful for reducingthe bandwidth for migration by excepting or excluding the copy forpage-unallocated area. The Volume Migration Control Program 103-9 on thefrontend storage subsystem 100 a orders the Disk Control Program 103-7to transfer the data from the external volume 231 and/or disks 106 tothe cache area 103-3. The Volume Migration Control Program 103-9 on thefrontend storage subsystem 100 a reads data from the cache area 103-0and transfers the data and configuration to the Volume Migration ControlProgram 103-9 on the frontend storage subsystem 100 b. The VolumeMigration Control Program 103-9 on the frontend storage subsystem 100 bprovisions a host volume 131 with the aforesaid configuration byapplying it to the Volume Management Table 103-9 and stores the data onthe cache area 103-0. The Disk Controls Program 103-7 allocates acapacity page by changing the Page Management Table 103-4. The DiskControls Program 103-7 stores data on the cache area 103-0 to theexternal volume 231 or disks 106 by resolving the mapping with the DiskManagement Table 103-1.

C9. Integrated Storage Pool Reduction

FIG. 50 shows an example of the sequence of reduction of the integratedstorage pool 600 by removal of a storage subsystem. In procedure S9-1,the user designates a storage subsystem to be removed from theintegrated storage pool. After the pool reduction process completes, theuser removes the storage from the system. In procedure S9-2, theintegrated storage pool management server 300 checks the configurationof the designated storage subsystem.

If the storage subsystem is a backend storage subsystem 200, thefollowing occurs. The frontend storage subsystem 100 which mounts theremoved backend storage subsystem 200 reallocates capacity pages fromthe external volumes 216 to other RAID groups 112. The frontend storagesubsystem 100 unmounts the external volume 216.

If the storage subsystem is a frontend storage subsystem 100, thefollowing occurs. The frontend storage subsystem 100 a copies the dataand configuration of the host volumes 131 to other frontend storagesubsystems 100. The host 500 which connects the copied host volumes 131changes the path to copy the target host volume 131. The frontendstorage subsystem 100 a deletes the copy-source host volumes 131. Thefrontend storage subsystem 100 b which does not include the removedfrontend storage subsystem 100 a reallocates capacity pages from theRAID groups mounting the cross mount volumes 132 on the removed frontendstorage subsystem 100. The frontend storage subsystem 100 b which doesnot include the removed frontend storage subsystem 100 a unmounts thecross mount volumes 132 of the removed frontend storage subsystem 100 a.The removed frontend storage subsystem 100 unmounts the external volumes231. The frontend storage subsystem 100 b which does not include theremoved frontend storage subsystem 100 a mounts the unmounted externalvolumes 231.

In procedure S9-3, the frontend storage subsystem 100 reads theconfiguration and data of all host volumes 131. In procedure S9-4, thehost volume 131 transfers the data. In procedure S9-5, the frontendstorage subsystem 100 a transfers the data and configuration of the hostvolume 131 to the frontend storage subsystem 100 a. In procedure S9-6,the host volume 131 on the frontend storage subsystem 100 b isprovisioned and stores the data. In procedure S9-7, the host 500 changesthe path from the frontend storage subsystem 100 a to the frontendstorage subsystem 100 b. In procedure S9-8, the host volume 131 on thefrontend storage subsystem 100 a disconnects the host path. In procedureS9-9, the host volume 131 on the frontend storage subsystem 100 bconnects the host path. In procedure S9-10, the frontend storagesubsystem 100 a deletes all host volumes 131. In procedure S9-11, thehost volume 131 is deleted. The procedure S9-12, the frontend storagesubsystem 100 b reallocates the capacity pages on the RAID groupmounting an external volume 231 to other RAID groups. In procedureS9-13, the RAID group copies data to other RAID groups 112. In procedureS9-14, the cross mount volume 132 changes the mapping for capacitypages. In procedure S9-15, the frontend storage subsystem 100 b removesthe RAID groups 112 mounting the cross mount volumes 132 from the tieredthin provisioning pool 110. The frontend storage subsystem 100 bunmounts the mounted cross mount volumes 132. In procedure S9-16, thecross mount volume 132 on the frontend storage subsystem 100 a isunmounted. In procedure S9-17, the frontend storage subsystem 100 aunmounts the external volumes 216. In procedure S9-18, the RAID group112 unmounts the external volume 216. In procedure S9-19, the frontendstorage subsystem 100 b mounts the external volume 216 which was mountedon the frontend storage subsystem 100 a. In procedure S9-20, the RAIDgroup 112 mounts the external volume 216. In procedure S9-21, thefrontend storage subsystem 100 a is removed from the system.

FIG. 51 shows the state of the system during the reduction process ofthe integrated storage pool 600. The frontend storage subsystem 100 amigrates a host volume 131 to the frontend storage subsystem 100 b. Thefrontend storage subsystem 100 b mounts an external volume 213 which thefrontend storage subsystem 100 a has mounted. The connection to thecross mount volume 132 of the frontend storage subsystem 100 a mountedby the frontend storage subsystem 100 b is disconnected. The connectionfrom the host 500 to the host volume 131 of the frontend storagesubsystem 100 a is succeeded to or switched to the frontend storagesubsystem 100 b.

Of course, the system configurations illustrated in FIGS. 1, 4, 10, 13,and 21 are purely exemplary of information systems in which the presentinvention may be implemented, and the invention is not limited to aparticular hardware configuration. The computers and storage systemsimplementing the invention can also have known I/O devices (e.g., CD andDVD drives, floppy disk drives, hard drives, etc.) which can store andread the modules, programs and data structures used to implement theabove-described invention. These modules, programs and data structurescan be encoded on such computer-readable media. For example, the datastructures of the invention can be stored on computer-readable mediaindependently of one or more computer-readable media on which reside theprograms used in the invention. The components of the system can beinterconnected by any form or medium of digital data communication,e.g., a communication network. Examples of communication networksinclude local area networks, wide area networks, e.g., the Internet,wireless networks, storage area networks, and the like.

In the description, numerous details are set forth for purposes ofexplanation in order to provide a thorough understanding of the presentinvention. However, it will be apparent to one skilled in the art thatnot all of these specific details are required in order to practice thepresent invention. It is also noted that the invention may be describedas a process, which is usually depicted as a flowchart, a flow diagram,a structure diagram, or a block diagram. Although a flowchart maydescribe the operations as a sequential process, many of the operationscan be performed in parallel or concurrently. In addition, the order ofthe operations may be re-arranged.

As is known in the art, the operations described above can be performedby hardware, software, or some combination of software and hardware.Various aspects of embodiments of the invention may be implemented usingcircuits and logic devices (hardware), while other aspects may beimplemented using instructions stored on a machine-readable medium(software), which if executed by a processor, would cause the processorto perform a method to carry out embodiments of the invention.Furthermore, some embodiments of the invention may be performed solelyin hardware, whereas other embodiments may be performed solely insoftware. Moreover, the various functions described can be performed ina single unit, or can be spread across a number of components in anynumber of ways. When performed by software, the methods may be executedby a processor, such as a general purpose computer, based oninstructions stored on a computer-readable medium. If desired, theinstructions can be stored on the medium in a compressed and/orencrypted format.

From the foregoing, it will be apparent that the invention providesmethods, apparatuses and programs stored on computer readable media fortiered storage pool management and control for lose coupled multiplestorage environment. Additionally, while specific embodiments have beenillustrated and described in this specification, those of ordinary skillin the art appreciate that any arrangement that is calculated to achievethe same purpose may be substituted for the specific embodimentsdisclosed. This disclosure is intended to cover any and all adaptationsor variations of the present invention, and it is to be understood thatthe terms used in the following claims should not be construed to limitthe invention to the specific embodiments disclosed in thespecification. Rather, the scope of the invention is to be determinedentirely by the following claims, which are to be construed inaccordance with the established doctrines of claim interpretation, alongwith the full range of equivalents to which such claims are entitled.

What is claimed is:
 1. A system comprising: a first storage systemincluding a first storage controller, which receives input/output (I/O)commands from host computers and provides first storage volumes to thehost computers; a second storage system including a second storagecontroller, which receives I/O commands from host computers andprovisions second storage volumes to the host computers; and a storagemedia which is allocated to a first pool; wherein a first data storingregion of one of the first storage volumes is allocated from the firstpool by the first storage controller, wherein a second data storingregion of one of the second storage volumes is allocated from a secondpool by the second storage controller, wherein in response to a commandfrom a management server, the second storage controller allocates thestorage media to the second pool and the first storage controllerreleases the storage media from the first pool.
 2. The system accordingto claim 1, further comprising: a management server, wherein themanagement server selects to migrate a first storage volume to a secondstorage volume, or change control from the second storage controller tothe first storage controller against a storage media which is coupled toboth the first and second storage controllers, based on load requirementof the system.